{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b4da38ad",
   "metadata": {},
   "source": [
    "# 1. Libraries & Sample Data\n",
    "The first step is to load our Python Libraries and download the sample data. The dataset represents Apple stock price (1d bars) for the year 2010"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1aa05430",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load Python Libraries\n",
    "import math\n",
    "import keras\n",
    "import random\n",
    "import datetime\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "from tqdm.notebook import tqdm\n",
    "from collections import deque\n",
    "from IPython.display import display, HTML\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# for dataframe display\n",
    "pd.set_option('display.max_rows', None)\n",
    "def display_df(df):\n",
    "    # Puts the scrollbar next to the DataFrame\n",
    "    display(HTML(\"<div style='height: 200px; overflow: auto; width: fit-content'>\" + df.to_html() + \"</div>\"))\n",
    "\n",
    "# for reproducability of training rounds\n",
    "keras.utils.set_random_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa831031",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download Sample Data\n",
    "data = pd.read_csv('GOOG_2009-2010_6m_RAW_1d.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c4a4a35",
   "metadata": {},
   "source": [
    "# 2. Exploratory Data Analysis\n",
    "Next, we want to analyze our data. Display the data as a dataframe, and plot some relevant data so you can get an idea of what our dataset looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aedbfa86",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display as Dataframe\n",
    "display_df(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b0f9772",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Index data by Date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b401f82f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the Close Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "189bc851",
   "metadata": {},
   "source": [
    "# 3. Data Cleaning\n",
    "Next, we need to clean our data for training our model. This requires removal of NaN values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0ebfafd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check for null values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9128668",
   "metadata": {},
   "outputs": [],
   "source": [
    "# forward fill missing values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a6050c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check for null values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "527f4d09",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the cleaned Close Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "764fed7f",
   "metadata": {},
   "source": [
    "# 4. Feature Selection\n",
    "Now that we have cleaned our stock data, we need to select which features to train our model on. For this project, we will be training with Close data and 20-day Bollinger Bands of Close."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dcb99ad6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate 20-day bollinger bands\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42224d3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove rows with NaN bollinger bands"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8f48c1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define new dataframe with only the training features (Close, Upper BB, Lower BB)\n",
    "dataset = # define df here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2f718a0",
   "metadata": {},
   "source": [
    "# 5. Normalization\n",
    "Now that we have cleaned our data, created our indicators of interest, and selected our features, we must normalize our data. For this project, we use the sklearn StandardScaler, which centers the data and normalizes to unit variance. We will not be using a rolling scaler for this project, due to the complexity of back-translating to true proce and indicator values - you can try this yourself once you have completed the project. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85eb7ff4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display & Plot Un-normalized Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1f2cbb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Normalize Dataset with StandardScaler\n",
    "normlist = []\n",
    "normed_dataset = pd.DataFrame(index=dataset.index)\n",
    "for col in dataset.columns:\n",
    "    normalizer = StandardScaler()\n",
    "    column_data = # define column data\n",
    "    # fit normalizer to column data\n",
    "    # transform column data with the fitted normalizer, and place the transformed data column in out normed_dataset df\n",
    "    # append the fitted normalizer to normlist for use later\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7dcab016",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display & Plot Normalized Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60e8526a",
   "metadata": {},
   "source": [
    "# 6. Train / Test Split\n",
    "Now that our data cleaned, features are selected, and the dataset is normalized, we are ready to feed the data into our model. With this in mind, we split the data ito train and test data (50/50 split)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7557879",
   "metadata": {},
   "outputs": [],
   "source": [
    "# split dataset df into train (50%) and test (50%) datasets\n",
    "\n",
    "train_df =  # define training dtaframe under this variable name\n",
    "test_df =  # define testing dtaframe under this variable name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1dca49e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# display train and test dfs (ensure no overlap)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92a6ed95",
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert train and test dfs to np arrays with dtype=float\n",
    "X_train = # define training array under this variable name\n",
    "X_test = # define testing array under this variable name\n",
    "# print the shape of X_train to remind yourself how many examples and features are in the dataset\n",
    "# track index to remember which feature is which\n",
    "idx_close = # numerical idx of close data column in array\n",
    "idx_bb_upper = # numerical idx of BB Upper data column in array\n",
    "idx_bb_lower = # numerical idx of BB Upper data column in array"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "714c77a8",
   "metadata": {},
   "source": [
    "# 7. Define the Agent\n",
    "Now that our data is ready to use, we can define the Reinforcement Learning Agent."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abd06b6d",
   "metadata": {},
   "source": [
    "### Define the DQN Model\n",
    "The first step in defining our agent is the Deep Q-Network model definition. For this project, we are creating a model sequential model with four layers. The first three layers have output shape of 64, 32, and 8, respectively, and a RELU activation. The output layer has an output shape of the size of our action space (buy, sell, hold), and a linear activation. Our Loss finction is Mean Squared Error, and our optimizer is Adam with a learning rate of 0.001. Use Keras to build this model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1e98591",
   "metadata": {},
   "outputs": [],
   "source": [
    "@keras.saving.register_keras_serializable()\n",
    "# Define DQN Model Architecture\n",
    "class DQN(keras.Model):\n",
    "    def __init__(self, state_size, action_size):\n",
    "\n",
    "        # define model layers in keras\n",
    "\n",
    "        # compile model in keras\n",
    "\n",
    "        # save model to DQN instance\n",
    "        self.model = model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edabcd7d",
   "metadata": {},
   "source": [
    "### Define Agent Class\n",
    "Now that we have defined our underlying DQN Model, we must define out Reinforcement Learning Agent. The agent initialization is provided for you, you must define an act function, and an expereince replay function. As a reminder, the act function defines how our model will act (buy, hold, or sell) given a certain state. The Experience Replay function tackles catastrophic forgetting in our training process, by maintaining a memory buffer to allow training on independent / randomized minibatches of previous states. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1489f4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Agent:\n",
    "    def __init__(self, window_size, num_features, test_mode=False, model_name=''):\n",
    "        self.window_size = window_size # How many days of historical data do we want to include in our state representation?\n",
    "        self.num_features = num_features # How many training features do we have?\n",
    "        self.state_size = window_size*num_features # State size includes number of training features per day, and number of lookback days \n",
    "        self.action_size = 3 # 0=hold, 1=buy, 2=sell\n",
    "        self.memory = deque(maxlen=1000) # Bound memory size: once the memory reaches 1000 units, the lefthand values are discarded as righthand values are added\n",
    "        self.inventory = [] # Inventory to hold trades\n",
    "        self.model_name = model_name # filename for saved model checkpoint loading\n",
    "        self.test_mode = test_mode # flag for testing (allows model load from checkpoint model_name)\n",
    "\n",
    "        self.gamma = 0.95\n",
    "        self.epsilon = 1.0\n",
    "        self.epsilon_min = 0.01\n",
    "        self.epsilon_decay = 0.995\n",
    "        \n",
    "        self.model = keras.models.load_model(model_name) if test_mode else self._model()\n",
    "\n",
    "    #Deep Q Learning (DQL) model\n",
    "    def _model(self):\n",
    "        model = DQN(self.state_size, self.action_size).model\n",
    "        return model\n",
    "    \n",
    "    # DQL Predict (with input reshaping)\n",
    "    #   Input = State\n",
    "    #   Output = Q-Table of action Q-Values\n",
    "    def get_q_values_for_state(self, state):\n",
    "        return self.model.predict(state.flatten().reshape(1, self.state_size))\n",
    "    \n",
    "\n",
    "    # DQL Fit (with input reshaping)\n",
    "    #   Input = State, Target Q-Table \n",
    "    #   Output = MSE Loss between Target Q-Table and Actual Q-Table for State\n",
    "    def fit_model(self, input_state, target_output):\n",
    "        return self.model.fit(input_state.flatten().reshape(1, self.state_size), target_output, epochs=1, verbose=0)    \n",
    "    \n",
    "    # Agent Action Selector\n",
    "    #   Input = State\n",
    "    #   Policy = epsilon-greedy (to minimize possibility of overfitting)\n",
    "    #   Intitially high epsilon = more random, epsilon decay = less random later\n",
    "    #   Output = Action (0, 1, or 2)\n",
    "    def act(self, state): \n",
    "        # Choose any action at random (Probablility = epsilon for training mode, 0% for testing mode)\n",
    "        if not self.test_mode and random.random() <= self.epsilon:\n",
    "            # **select random action here**\n",
    "        # Choose the action which has the highest Q-value (Probablitly = 1-epsilon for training mode, 100% for testing mode)\n",
    "        # **use model to select action here - i.e. use model to assign q-values to all actions in action space (buy, sell, hold)**\n",
    "        # **return the action that has the highest value from the q-value function.**\n",
    " \n",
    "    # Experience Replay (Learning Function)\n",
    "    #   Input = Batch of (state, action, next_state) tuples\n",
    "    #   Optimal Q Selection Policy = Bellman equation\n",
    "    #   Important Notes = Model fitting step is in this function (fit_model)\n",
    "    #                     Epsilon decay step is in this function\n",
    "    #   Output = Model loss from fitting step\n",
    "    def exp_replay(self, batch_size, losses):\n",
    "        losses = []\n",
    "        # define a mini-batch which holds batch_size most recent previous memory steps (i.e. states)\n",
    "        for state, action, reward, next_state, done in mini_batch:\n",
    "            # reminders: \n",
    "            #   - state is a vector containing close & MA values for the current time step\n",
    "            #   - action is an integer representing the action taken by the act function at the current time step- buy, hold, or sell\n",
    "            #   - reward represents the profit of a given action - it is either 0 (for buy, hold, and sells which loose money) or the profit in dollars (for a profitable sell)\n",
    "            #   - next_state is a vector containing close & MA values for the next time step\n",
    "            #   - done is a boolean flag representing whether or not we are in the last iteration of a training episode (i.e. True when next_state does not exist.)\n",
    "            if done:\n",
    "                # special condition for last training epoch in batch (no next_state)\n",
    "                optimal_q_for_action = reward  \n",
    "            else:\n",
    "                # target Q-value is updated using the Bellman equation: reward + gamma * max(predicted Q-value of next state)\n",
    "                optimal_q_for_action = #  reward + gamma * max(predicted Q-value of next state)\n",
    "            # Get the predicted Q-values of the current state\n",
    "            # Update the output Q table - replace the predicted Q value for action with the target Q value for action \n",
    "            # Fit the model where state is X and target_q_table is Y\n",
    "            history = self.fit_model(#X, Y)\n",
    "            losses += history.history['loss']\n",
    "           \n",
    "        # define epsilon decay (for the act function)\n",
    "        if self.epsilon > self.epsilon_min:\n",
    "            self.epsilon *= self.epsilon_decay\n",
    "        return losses"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfee09a6",
   "metadata": {},
   "source": [
    "# 8. Train the Agent\n",
    "Now that our data is ready and our agent is defined, we are ready to train the agent. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a05cd0f4",
   "metadata": {},
   "source": [
    "### Helper Functions\n",
    "Before we define the training loop, we will write some helper functions: one for printing price data, one to define the sigmoid funtion, one to grab the state representation,  one to plot the trading output of our trained model, and one to plot the training loss. The printing, sigmoid, and plotting functions are defined for you. You must define the function which gets the state representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "722fbbdb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Format price string\n",
    "def format_price(n):\n",
    "    return ('-$' if n < 0 else '$') + '{0:.2f}'.format(abs(n))\n",
    "\n",
    "def sigmoid(x):\n",
    "    return 1 / (1 + math.exp(-x))\n",
    "\n",
    "# Plot behavior of trade output\n",
    "def plot_behavior(data_input, bb_upper_data, bb_lower_data, states_buy, states_sell, profit, train=True):\n",
    "    fig = plt.figure(figsize = (15,5))\n",
    "    plt.plot(data_input, color='k', lw=2., label= 'Close Price')\n",
    "    plt.plot(bb_upper_data, color='b', lw=2., label = 'Bollinger Bands')\n",
    "    plt.plot(bb_lower_data, color='b', lw=2.)\n",
    "    plt.plot(data_input, '^', markersize=10, color='r', label = 'Buying signal', markevery = states_buy)\n",
    "    plt.plot(data_input, 'v', markersize=10, color='g', label = 'Selling signal', markevery = states_sell)\n",
    "    plt.title('Total gains: %f'%(profit))\n",
    "    plt.legend()\n",
    "    if train:\n",
    "        plt.xticks(range(0, len(train_df.index.values), int(len(train_df.index.values)/15)), train_df.index.values[0:: int(len(train_df.index.values)/15)], rotation=45, fontsize='small')\n",
    "    else:\n",
    "        plt.xticks(range(0, len(test_df.index.values), int(len(test_df.index.values)/15)), test_df.index.values[0::int(len(test_df.index.values)/15)], rotation=45, fontsize='small')\n",
    "    plt.show()\n",
    "\n",
    "# Plot training loss\n",
    "def plot_losses(losses, title):\n",
    "    plt.plot(losses)\n",
    "    plt.title(title)\n",
    "    plt.ylabel('MSE Loss Value')\n",
    "    plt.xlabel('batch')\n",
    "    plt.show()\n",
    "\n",
    "# returns an an n-day state representation ending at time t\n",
    "def get_state(data, t, n): \n",
    "    # data is the dataset of interest which holds the state values (i.e. Close , BB Upper, BB Lower)\n",
    "    # t is the current time step \n",
    "    # n is the size of the training window \n",
    "\n",
    "    # the first step is to get the window of the dataset at the current time step (eg. if window size is 1, we grab the previous and the current time step)\n",
    "    # remember to define the special case for the first iteration, where there is no previous time step. See lesson X for a reminder of how to do this.\n",
    "\n",
    "    # once we have our state data, we need to apply the sigmoid to each feature.\n",
    "    # return an array holding the n-day sigmoid state representation\n",
    "    return np.array([res])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e6bd8f6",
   "metadata": {},
   "source": [
    "### Training Loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "798817f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# display the shape of your training data in order to remond yourself how may features and examples there are in your training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7780da37",
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.utils.disable_interactive_logging()  # disable built-in keras loading bars - they make the output difficult to read and monitor\n",
    "\n",
    "window_size = 1\n",
    "\n",
    "agent = # instatniate the agent using the wndow size and the number of training features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "851650c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.config.disable_traceback_filtering() # disable built-in keras loading bars - they make the output difficult to read and monitor\n",
    "\n",
    "\n",
    "l = # track number of examples in dataset (i.e. number of days to train on)\n",
    "\n",
    "# batch size defines how often to run the exp_replay method\n",
    "batch_size = 32\n",
    "\n",
    "#An episode represents a complete pass over the data.\n",
    "episode_count = 2\n",
    "\n",
    "normalizer_close = # get the close normalizer from normlist\n",
    "normalizer_bb_upper = # get the BB upper normalizer from normlist\n",
    "normalizer_bb_lower = # get the BB lower normalizer from normlist\n",
    "\n",
    "X_train_true_price = # inverse transform the Close column in X_train in order to get true close prices\n",
    "X_train_true_bb_upper = # inverse transform the Upper BB column in X_train in order to get true Upper BB values\n",
    "X_train_true_bb_lower = # inverse transform the Lower BB column in X_train in order to get true Lower BB values\n",
    "\n",
    "batch_losses = []\n",
    "num_batches_trained = 0\n",
    "\n",
    "for e in range(episode_count + 1):\n",
    "    state = # get the state for the first step\n",
    "    # initialize variables\n",
    "    total_profit = 0\n",
    "    total_winners = 0\n",
    "    total_losers = 0\n",
    "    agent.inventory = []\n",
    "    states_sell = []\n",
    "    states_buy = []\n",
    "    for t in tqdm(range(l), desc==f'Running episode {e}/{episode_count}'):\n",
    "        # get the action\n",
    "        # get the next state\n",
    "        \n",
    "        # initialize reward for the current time step\n",
    "        reward = 0\n",
    "\n",
    "        if action == 1: # buy\n",
    "            # inverse transform to get true buy price in dollars\n",
    "            # append the buy price to the inventory\n",
    "            # append the time step to states_buy\n",
    "            # print the action and price of the action\n",
    "\n",
    "        elif action == 2 and len(agent.inventory) > 0: # sell\n",
    "            # get the bought price of the stock you are selling (i.e. the stock at the beginning of the inventory)\n",
    "            bought_price = agent.inventory.pop(0)  \n",
    "            # inverse transform to get true sell price in dollars\n",
    "            # define reward as max of profit (close price at time of sell - close price at time of buy) and 0\n",
    "            # add current profit to total profit\n",
    "            if trade_profit >=0:\n",
    "                # add current profit to total winners\n",
    "            else:\n",
    "                # add current profit to total losers\n",
    "            # append the time step to states_sell\n",
    "            # print the action, price of the action, and profit of the action\n",
    "\n",
    "        # flag for final training iteration\n",
    "        done = True if t == l - 1 else False\n",
    "\n",
    "        # append the details of the state action etc in the memory, to be used by the exp_replay function\n",
    "        state = next_state\n",
    "\n",
    "        # print total profit and plot behaviour of the current episode when the episode is finished\n",
    "        if done:\n",
    "            print('--------------------------------')\n",
    "            print(f'Episode {e}')\n",
    "            print(f'Total Profit: {format_price(total_profit)}')\n",
    "            print(f'Total Winners: {format_price(total_winners)}')\n",
    "            print(f'Total Losers: {format_price(total_losers)}')\n",
    "            print(f'Max Loss: {max(batch_losses[num_batches_trained:len(batch_losses)])}')\n",
    "            print(f'Total Loss: {sum(batch_losses[num_batches_trained:len(batch_losses)])}')\n",
    "            print('--------------------------------')\n",
    "            plot_behavior(X_train_true_price, X_train_true_bb_upper, X_train_true_bb_lower, states_buy, states_sell, total_profit)\n",
    "            plot_losses(batch_losses[num_batches_trained:len(batch_losses)], f'Episode {e} DQN model loss')\n",
    "            num_batches_trained = len(batch_losses)\n",
    "\n",
    "        # when the size of the memory is greater than the batch size, run the exp_replay function on the batch to fit the model and get losses for the batch\n",
    "        # then sum the losses for the batch and append them to the batch_losses list\n",
    "            \n",
    "    if e % 2 == 0:\n",
    "        # save the model every 2 episodes (in case of crash or better training iteration in the middle of training process)\n",
    "        agent.model.save(f'model_ep{e}.keras')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f599ee50",
   "metadata": {},
   "source": [
    "### Plot Training Loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54a03b09",
   "metadata": {},
   "outputs": [],
   "source": [
    "# use the plot_losses function to plot all batch_losses for the entire training round"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8adb831d",
   "metadata": {},
   "source": [
    "# 9. Test the trained agent \n",
    "Finally, we get to test our trained model to see how well it performs in our test set. Using the training loop above, define a method to run our trained model on our X_test dataset. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cbd7885",
   "metadata": {},
   "source": [
    "### Define Parameters\n",
    "Some test parameters are defined for you below. Fill out the missing data. If you need a hint, look up at the training loop. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02b3f476",
   "metadata": {},
   "outputs": [],
   "source": [
    "l_test = len(X_test) - 1\n",
    "state = get_state(X_test, 0, window_size + 1)\n",
    "total_profit = 0\n",
    "done = False\n",
    "states_sell_test = []\n",
    "states_buy_test = []\n",
    "\n",
    "#Get the trained model\n",
    "agent = Agent(window_size, num_features=X_test.shape[1], test_mode=True, model_name=f'model_ep{episode_count}.keras')\n",
    "agent.inventory = []\n",
    "\n",
    "state = # get the first state of the test dataset\n",
    "\n",
    "X_test_true_price = # true close price\n",
    "X_test_true_bb_upper = # true BB upper\n",
    "X_test_true_bb_lower = # true BB lower"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8871f51",
   "metadata": {},
   "source": [
    "### Run the Test\n",
    "Run the test data through the trained model. Look at the training loop for a hint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b474ba5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "for t in range(l_test):\n",
    "    action = agent.act(state)\n",
    "    next_state = # get the next state in the test dataset\n",
    "    reward = 0\n",
    "\n",
    "    if action == 1: # buy\n",
    "        # inverse transform to get true buy price in dollars\n",
    "        # append buy prive to inventory\n",
    "        # append time step to states_buy_test\n",
    "        print(f'Buy: {format_price(buy_price)}')\n",
    "\n",
    "    elif action == 2 and len(agent.inventory) > 0: # sell\n",
    "        # get bought price from beginning of inventory\n",
    "        # inverse transform to get true sell price in dollars\n",
    "        # reward is max of profit (close price at time of sell - close price at time of buy)\n",
    "        # update total_test_profit\n",
    "        # append time step to states_sell_test\n",
    "        print(f'Sell: {format_price(sell_price)} | Profit: {format_price(sell_price - bought_price)}')\n",
    "\n",
    "    \n",
    "    if t == l_test - 1:\n",
    "        done = True\n",
    "    # append to memory so we can re-train on 'live' (test) data later    \n",
    "    agent.memory.append((state, action, reward, next_state, done))\n",
    "    state = next_state\n",
    "\n",
    "    if done:\n",
    "        print('------------------------------------------')\n",
    "        print(f'Total Profit: {format_price(total_profit)}')\n",
    "        print('------------------------------------------')\n",
    "        \n",
    "plot_behavior(X_test_true_price, X_test_true_bb_upper, X_test_true_bb_lower, states_buy_test, states_sell_test, total_profit, train=False)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "rl-project-udacity",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
